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Turn display on, using the image 
from the light sensor, find display 
area based an the characteristics 
of the display bf the known image 
on the display. 



Turn display off, with the light sensor, capture a frame of the display 



Turn display on, capture a frame of the display sf 




Define the borders of the display in the image captured by the light sensor as continuous lines a 



Borders of the display as it is 
captured by the light sensor, their 
visual characteristics, location and 
curvature 



Continue to pointing device /■ ? 
registration ( Figure 1 5 ) 
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Instruct user to register the pointing device he/she will use. The user may select a pointing device from a list or allow tr 
system to view the pointing device by allowing the pointing device to be viewed by the light sensor. 
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Instruct user to point the-pointing 
device to various points in the 
display. Capture one or more 
frames of the display space with 
the pointing device 



Capture a frame of the display space without the pointing device 



Capture a frame of the display space with the pointing device 



Locate pointing device by examining the difference between the tv 




Determine the distinguishing characteristics of the pointing device as opposed to the rest of the display by observing it via 
the light sensor against any image on the display or against a set of calibration images. Adjust the light sensor position and 
sensing parameters for optimum operation. 
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Continue to computing the mapping 
between the display space registered 
by the light sensor and the computer 
display 
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Distinguishing characteristics 
of the pointing device against 
a variety of display 
backgrounds. 
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Turn display on, instruct the user to 
point the pointing device to a 
convenient location of the display 
(e.g., center). Using the image 
from the light sensor, find the 
pointing device based on the 
known characteristics of the 
pointing devipe. 



Instruct user to hide pointing device, with the light sensor, capture a 
frame of the display space 



Instruct user to point the pointing device to a convenient location of the 
display, capture a frame of the display space 



Locate the pointing device by examining the difference between tr 
two frames 



Adjust the light sensor position and sensing parameters for best viewing conditions 




Instruct user to point with the pointing device to the borders and/or various locations of the display. Capture frames with light sensor. 



Define the borders of the display space in the image captured by the tight sf 



>r as continuous lines or curves 



no 



Continue to pointing device 
registration ( Figure 17) 
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Borders of the display as it is 
captured by the light sensor, their 
visual characteristics, location and 
curvature 
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o point with the pointing device to the borders and/or various locations of the display. Capture frames with light sensor. 



Adjust the light sensor position and sensing parameters for best viewing conditions 




Determine the distinguishing characteristics of the pointing device as opposed to the rest of the display by observing it via 
the light sensor against the background of the virtual display Adjust the light sensor position and 

sensing parameters for optimum operation. 



Continue to computing the mapping 
between the display space registered 
by the light sensor and the computer 
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Distinguishing characteristics 
of the pointing device against 
a variety of display 
backgrounds. 
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Borders of the display as it is 
captured by the light sensor, 
their visual characteristics, 
location and curvature 
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Divide the display space into the same number 
of regions as the those of the computer display 
using the information from the borders of the 
display space. 



Distinguishing characteristics 
of the pointing device (against 
a variety of display 
backgrounds if real). 
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Establish correspondence between the display 
space observed by the light sensor and the 
computer display region by region. Make 
necessary adjustments to the boundapes of 
individual regions as necessary. 



Images captured with the 
pointing device pointing to 
various regions of the display. 



Using the information from the position of the 
painting device previously registered by the 
user and the images captured when the user 
pointed the pointing device to the regions of the 
display as instructed by the system, make 
further adjustments to the mapping between 
the image space registered by the light sensor 
and the computer display. 



Continue to system operation 



Z Mapping between the display 
space and the computer 
display 
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Define region around item or icon that 
will be used to determine if a single 
click is warranted 
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Define number of frames or length of 
time that the pointer has to be in the 
region to highlight the item 
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Highlight item. 



Find pointer device and position the 
computer mouse accordingly on the 
display, store mouse position is stack 




Throw out the oldest mouse 
coordinate from the stack. 
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System 
Operation 



Get data from sensor or 
image frame from camera 



Locate pointing device 




Map position of pointing 
device to point on the real 
or virtual display space 



Find the computer display 
position corresponding to 
the pointing device position 



Position the computer's 
pointing icon ("e.g., mouse 
arrow) at the computed 
computer display position 
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